Kaggle Titanic Competition - Data Exploration & Model Building

Objectives

1. End to end analysis using R
2. Learn the caret package for ML
3. Learn to present the case using R Notebooks

---

Read in the dataset

I stored the raw files on Github, so I used RCurl with Wehrley’s method that utilizes read.csv to the fullest. It’s one of the best ways I’ve found to read in data and also set data-types at the same time. He’s done a great job on that function. The dataset contains one ID variable, one response variable and ten predictor variables.

library(RCurl,quietly = T)
library(tidyverse,quietly = T)
library(ggplot2,quietly = T)
library(gridExtra,quietly = T)
library(Amelia,quietly = T)
library(beanplot,quietly = T)
library(caret,quietly = T)
library(stringr,quietly = T)
library(party, quietly = T)
# library(rattle, quietly = T)
readData <- function(path.name, file.name, column.types, missing.types) {
    gurl <- paste(path.name,file.name,sep="")
    download.file(gurl,file.name,method="curl",quiet = T)
    tbl_df(read.csv(file.name,colClasses=column.types,
             na.strings=missing.types))
}
Titanic.path <- "https://raw.githubusercontent.com/rsangole/Titanic/master/"
train.data.file <- "train.csv"
test.data.file <- "test.csv"
missing.types <- c("NA", "")
train.column.types <- c('integer',   # PassengerId
                        'factor',    # Survived
                        'factor',    # Pclass
                        'character', # Name
                        'factor',    # Sex
                        'numeric',   # Age
                        'integer',   # SibSp
                        'integer',   # Parch
                        'character', # Ticket
                        'numeric',   # Fare
                        'character', # Cabin
                        'factor'     # Embarked
)
test.column.types <- train.column.types[-2]     # # no Survived column in test.csv
train.raw <- readData(Titanic.path, train.data.file,train.column.types,missing.types)
test.raw <- readData(Titanic.path, test.data.file,test.column.types,missing.types)
prep_data <- function(D) {
    if (!is.null(D$Survived)) {
        D$Survived <- factor(D$Survived,
                             levels = c(1, 0),
                             labels = c('Survived', 'Dead'))
        }
    D$Pclass <- factor(D$Pclass,
                       levels = c(1, 2, 3),
                       labels = c('P1', 'P2', 'P3'))
    D$PassengerId <- NULL
    D
}
train.raw <- prep_data(train.raw)
test.raw <- prep_data(test.raw)
str(train.raw)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   891 obs. of  11 variables:
 $ Survived: Factor w/ 2 levels "Survived","Dead": 2 1 1 1 2 2 2 2 1 1 ...
 $ Pclass  : Factor w/ 3 levels "P1","P2","P3": 3 1 3 1 3 3 1 3 3 2 ...
 $ Name    : chr  "Braund, Mr. Owen Harris" "Cumings, Mrs. John Bradley (Florence Briggs Thayer)" "Heikkinen, Miss. Laina" "Futrelle, Mrs. Jacques Heath (Lily May Peel)" ...
 $ Sex     : Factor w/ 2 levels "female","male": 2 1 1 1 2 2 2 2 1 1 ...
 $ Age     : num  22 38 26 35 35 NA 54 2 27 14 ...
 $ SibSp   : int  1 1 0 1 0 0 0 3 0 1 ...
 $ Parch   : int  0 0 0 0 0 0 0 1 2 0 ...
 $ Ticket  : chr  "A/5 21171" "PC 17599" "STON/O2. 3101282" "113803" ...
 $ Fare    : num  7.25 71.28 7.92 53.1 8.05 ...
 $ Cabin   : chr  NA "C85" NA "C123" ...
 $ Embarked: Factor w/ 3 levels "C","Q","S": 3 1 3 3 3 2 3 3 3 1 ...

---

Missing values analysis

Quick investigation of missing values can be done using the complete.cases(), and more thorough graphical summary can be done using Amelia. Overall, 79% of the observations have some missing data.

#Complete cases (percentages)
round(prop.table(table(complete.cases(train.raw))),2)

FALSE  TRUE 
 0.79  0.21 

Amelia lets us graphically investigate which variables have missing data. purr::map_xxx() gives this same information numerically in a succint fashion.

missmap(train.raw, main='Missing Values Analysis using Amelia ordered by % missing', col=c('red', 'gray'),legend = F,rank.order = T)

#Missing cases (numbers):
map_int(train.raw,~sum(is.na(.x)))

Survived   Pclass     Name      Sex      Age    SibSp    Parch   Ticket     Fare    Cabin 
       0        0        0        0      177        0        0        0        0      687 
Embarked 
       2 

#Missing cases (percentages):
round(map_dbl(train.raw,~sum(is.na(.x))/length(.x)),2)

Survived   Pclass     Name      Sex      Age    SibSp    Parch   Ticket     Fare    Cabin 
    0.00     0.00     0.00     0.00     0.20     0.00     0.00     0.00     0.00     0.77 
Embarked 
    0.00 

Cabin has a large number of missing values (77% missing). Imputing this variable may prove challenging or even useless. Age (19.9% missing) and Embarked (0.2%) missing are much more managable.

---

EDA

The first step in the analysis is to explore the data numerically and graphically. I always split up my EDA investigation as follows:

• Target Variable
• Predictor Variables
  • Univariate
  • Bivariate
  • Multivariate

This gives me a structured approach towards larger datasets. My professor at Northwestern taught me to always complete a thorough intimate numeric & graphical EDA on the data, no matter how large the data ¹. Anscombe (1973) clearly shows the importance of graphical analyses.

Target Variable

Survived is the response variable. As we can see, a large majority of the passengers did not survive the accident. The response variable is a False/True boolean variable. Thus, the analysis techniques used later will be those appropriate for classification problems.

round(prop.table(table(train.raw$Survived)),2)

Survived     Dead 
    0.38     0.62 

---

Predictor Variables

Univariate & Bivariate

The first step is to look at every variable available. I prefer using the ggplot2 framework for all the visuals.

Continuous Variables

• Age seems to have a bimodal distribution - very young children, and then directly young adults to mid-age persons. The 2nd mode is right skewed with no obvious outliers.

• Fare certainly shows many outliers beyond the ~$200 level. A majority of the fares are <$50, which makes sense since a majority of the travelers are bound to be in the 3rd passenger class.

p1 <- ggplot(data=train.raw,aes(x=Age))+geom_histogram(bins = 40)
p2 <- ggplot(data=train.raw,aes(x=Fare))+geom_histogram(bins = 40)
grid.arrange(p1,p2)

As we can see, the median fare is $14.5, the mean is $32, but the max is $512. We’ll investigate winzorising this variable in the latter part. Perhaps a transformation will also help?

summary(train.raw$Fare)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.00    7.91   14.45   32.20   31.00  512.33 

Categorical Variables

A ggplot command is iterated over for the categorical variables.²

Key takeways for the categorical variables:

1. Pclass: If you were traveling 1st class, you have the highest chance of survival. Could be indicative of preferential treatment to those who paid more, a less politically correct class-stratified society, as well as the fact that the 1st class passengers had cabins at the very top of the ship.
2. Pclass: Persons traveling 3rd class had the highest fatality rate. 3rd class passengers had cabins deep in the ship. With the reasons give in (1), this could have contributed to the low survival rate.
3. Sex: Males have a very high fatality rate. Seems like the ‘women and children’ first policy was followed during evacuation.
4. SibSp & Parch: What’s interesting here is, for both these variables, at level 0, the fatality rate is higher. At levels 1+, the chances of survival are much better. Again, this could point to the ‘women and children’ policy being followed. (Or perhaps there weren’t as many families with children on board!)
5. Embarked: Southampton has a higher fatality rate than Cherbourg or Queenstown. A cross-tabulation between Embarked and Pclass shows that 72% of the 3rd class passengers and 89% of the 2nd class passengers boarded at Southampton. This jives with the observation that 2nd and 3rd class passengers have higher fatality rates.

get_legend<-function(myggplot){
  tmp <- ggplot_gtable(ggplot_build(myggplot))
  leg <- which(sapply(tmp$grobs, function(x) x$name) == "guide-box")
  legend <- tmp$grobs[[leg]]
  return(legend)
}
p <- lapply(X = c('Pclass','Sex','SibSp','Parch','Embarked'),
            FUN = function(x) ggplot(data = train.raw)+
                aes_string(x=x,fill='Survived')+
                geom_bar(position="dodge")+
                theme(legend.position="none"))
legend <- get_legend(ggplot(data = train.raw,aes(x=Pclass,fill=Survived))+geom_bar())
grid.arrange(p[[1]],p[[2]],p[[3]],p[[4]],p[[5]],legend,layout_matrix =
                 cbind(c(1,2,3),c(4,5,NA),c(6,6,6)),widths=c(3,3,1))

# round(prop.table(table(train.raw$Embarked,train.raw$Pclass),margin = 2),2)

Multivariate Analyses

Grouped boxplots are a common method of comparing distributions grouped by categorical variables. I find beanplots to be excellent complementary plots to boxplots (and in some cases, even better). They’re a bit tricky to read at first - since they are so underutilized - but just through one plot, a wealth of information can be extracted.³

Here is a comparison of the same information between a boxplot and a beanplot. What can we infer from the bean plot better?

1. The beanplot allows us to visualize the density function of the parameter, in this case: Age. Furthermore, the length of each beanline is cumulative to the number of datapoints that exist. Rightaway, we can tell that Pclass=3 has the most data in the set, with sparser data at Pclass=1.
2. The mean values for 1st class is higher than that for 2nd and 3rd class. The distributions of deceased and survived for 1st class are fairly similar.
3. For 2nd and 3rd class, the survived data shows a bimodal distribution. Bumps at the 0-10 age show that children were evacuated first. This is also the reason the mean values for survived is lower.
4. For 2nd and 3rd class, the deceased data shows a fairly normal distribution.
5. The individual measurements (represented by black lines) represent each observation and help identify outliers much more easily than a boxplot does.

ggplot(train.raw,aes(y=Age,x=Pclass))+geom_boxplot(aes(fill=Survived))+theme_bw()

beanplot(Age~Survived*Pclass,side='b',train.raw,col=list('yellow','orange'),
         border = c('yellow2','darkorange'),ll = 0.05,boxwex = .5,
         main='Passenger survival by pclass and Age',xlab='Passenger Class',ylab='Age')
legend('topright', fill = c('yellow','orange'), legend = c("Dead", "Survived"),bty = 'n',cex = .8)

A look into the SibSp and Parch variables shows something interesting. There are three regions one can identify:

• The probability of survival is minimal for number of parents/children aboard > 3.
• The probability of survival is minimal for number of siblings/spouses aboard > 3.
• For SibSp<=3 and Parch<=3, there are better chances for survival.

The grouping by Pclass reveals that all the large families were 3rd class travelers. Worse access to help… lowest chance for survival.

These could be simple rules either hard coded during model building: something along the lines of: IF (SibSp>3 OR Parch >3) THEN prediction = 0, or some derived variables can be created.

ggplot(train.raw,aes(y=SibSp,x=Parch))+
    geom_jitter(aes(color=Survived,shape=Pclass))+
    theme_bw()+
    scale_shape(solid=F)+
    geom_vline(xintercept = 3,color='darkblue',lty=3)+
    geom_hline(yintercept = 3,color='darkblue',lty=3)

---

Data Preparation

Missing Values Imputation

Starting with the easier one first:

Embarked: The largest portion of the passengers embared at Southhampton. I’m replacing the NAs with the same. First, I create a new imputed training dataset.

summary(train.raw$Embarked)

   C    Q    S NA's 
 168   77  644    2 

train.imp <- train.raw
train.imp$Embarked[is.na(train.imp$Embarked)]='S'

Names, Titles & Age:

The names have titles embedded in the strings. I can extract these using regex. Master, Miss, Mr and Mrs are the most popular - no surprise there, with lots of other titles. Here’s the distribution of the titles by age. These can be used to impute the missing age values.

train.raw$title <- str_extract(pattern = '[a-zA-Z]+(?=\\.)',string = train.raw$Name)
train.raw$title <- as.factor(train.raw$title)
train.raw %>%
    na.omit() %>%
    group_by(title) %>%
    dplyr::summarise(Count=n(), Median_Age=round(median(Age),0)) %>%
    arrange(-Median_Age)

ggplot(train.raw,aes(x=title,y=Age))+
    stat_summary(aes(y = Age,group=1), fun.y=median, colour="red", geom="point",group=1)+
    geom_jitter(shape=21,alpha=.6,col='blue')+
    theme_bw()+
    theme(axis.text.x = element_text(angle = 45, hjust = 1),legend.position="none")+
    labs(caption='Red points are median values')

Grouping similar titles together, I’ve kept a few titles - Officer, Royalty, Mr, Mrs and Miss.

train.imp <- train.raw
train.imp$title <- as.character(train.imp$title)
train.imp$title[train.imp$title %in% c('Capt','Col','Major')] <- 'Officer'
train.imp$title[train.imp$title %in% c('Don','Dr','Rev','Sir','Jonkheer','Countess','Lady','Dona')] <- 'Royalty'
train.imp$title[train.imp$title %in% c('Mrs','Mme')] <- 'Mrs'
train.imp$title[train.imp$title %in% c('Ms','Mlle')] <- 'Miss'
train.imp$title <- as.factor(train.imp$title)
train.imp %>%
    group_by(title) %>%
    summarise(Median_Age=median(Age,na.rm = T))

ggplot(train.imp,aes(x=title,y=Age))+
    geom_jitter(shape=21,alpha=.6,col='blue')+
    stat_summary(aes(y = Age,group=1), fun.y=median, colour="red", geom="point",group=1)+
    theme_bw()+
    theme(axis.text.x = element_text(angle = 45, hjust = 1),legend.position="none")+
    labs(caption='Red points are median values')

Now for the missing Age values. I’m trying out two strategies to impute age, just for kicks. First, a regression tree using the rpart method. 5-repeat 10-fold cross validation across a tuning grid of 20 values of maxdepth. RMSE stablizes at a depth of 14, with a value of 12.2.

age.predictors <- train.imp %>%
    dplyr::select(-Survived,-Cabin,-Ticket,-Name) %>%
    filter(complete.cases(.))
set.seed(1234)
ctrl <- trainControl(method = "boot",
                     repeats = 5,
                     number = 200
                     )
rpartGrid <- data.frame(maxdepth = seq(4,20,2))
rpartFit <- train(Age~.,
                  data=age.predictors,
                  method='rpart2',
                  trControl = ctrl,
                  tuneGrid = rpartGrid
                  )
rpartFit

CART 

712 samples
  7 predictor

No pre-processing
Resampling: Bootstrapped (200 reps) 
Summary of sample sizes: 712, 712, 712, 712, 712, 712, ... 
Resampling results across tuning parameters:

  maxdepth  RMSE      Rsquared 
   4        12.91352  0.2172555
   6        12.56362  0.2600303
   8        12.37466  0.2835666
  10        12.28184  0.2961068
  12        12.23967  0.3028092
  14        12.23329  0.3046570
  16        12.24043  0.3041673
  18        12.23669  0.3045630
  20        12.23821  0.3044234

RMSE was used to select the optimal model using  the smallest value.
The final value used for the model was maxdepth = 14.

plot(rpartFit)

plot(rpartFit$finalModel,margin=0.02)
text(rpartFit$finalModel,cex=0.8)

Another way is to run a randomforest with a search over values of mtry using 5-repeat 10-fold cross validation. As we can see mtry=4 is the optimal value which results in the lowest RMSE of 11.4; much better than the rpart model.

set.seed(1234)
rfGrid <- data.frame(mtry=seq(1,6,1))
ctrl <- trainControl(method = "repeatedcv",
                     repeats = 5
                     )
rfFit <- train(Age~.,
                  data=age.predictors,
                  method='rf',
                  trControl = ctrl,
                  tuneGrid = rfGrid)
rfFit

Random Forest 

712 samples
  7 predictor

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 642, 640, 642, 641, 640, 639, ... 
Resampling results across tuning parameters:

  mtry  RMSE      Rsquared 
  1     12.46449  0.3816241
  2     11.33714  0.4192503
  3     11.05166  0.4263448
  4     11.04766  0.4217797
  5     11.10717  0.4152716
  6     11.20324  0.4066238

RMSE was used to select the optimal model using  the smallest value.
The final value used for the model was mtry = 4.

plot(rfFit)

I’m going to use the randomForest model. Using the predict.train() to predict values of age and plug them back into the imputed data. You can see the blue points which are the imputed values of Age. What I noticed is that for all the titles, the imputed Age value seems to be distributed fairly well, except Master. For Master, the three imputed are definitely outliers. I’m going to force these to the median Age.

missing.age <- train.imp %>% filter(is.na(Age))
age.predicted <- predict(rfFit, newdata = missing.age)
train.imp %>%
    mutate(AgeMissing = is.na(Age),
           Age = ifelse(AgeMissing,age.predicted,Age)) %>%
    ggplot(aes(x=title,y=Age))+
    stat_summary(aes(y = Age,group=1), fun.y=median, colour="red", geom="point",group=1)+
    geom_jitter(aes(y=Age,col=AgeMissing),shape=2)+
    theme_bw()+
    theme(axis.text.x = element_text(angle = 45, hjust = 1),legend.position="none")+
    labs(caption='Red points are median values')

train.imp$Age[is.na(train.imp$Age)] <- age.predicted
train.imp$Age[train.imp$title=='Master' & train.imp$Age > 20] <- median(train.imp$Age[train.imp$title=='Master'],na.rm = T)

Derived Variables

Child?: Trying out two engineered variables here - is the passenger a child or not? Using Age=18 as a threshold. And is s/he close enough to be considered a adult by chance? Those between 16 and 18 could be mistaken for not being children. (My way of incorporating a fudge factor in the decision process of ladies & children first.)

train.imp$child <- 0
train.imp$child[train.imp$Age<18] <- 1
train.imp$almostadult <- as.numeric(between(train.imp$Age,16,18))

Really young, or really old?: Really young ones and older folks would get priority perhaps. Creating two categorical binary variables for these conditions.

train.imp$Young <- ifelse(train.imp$Age<10,1,0)
train.imp$Seniors <- ifelse(train.imp$Age>60,1,0)

Family related: Let’s also create some variables that talk about family sizes. What’s the total family size – continous variable TotalFam. Is the person single, part of a couple or a family? Three categorical variables for these.

train.imp$TotalFam <- train.imp$SibSp + train.imp$Parch + 1
# train.imp$LastName <- train.imp$Name %>% str_extract(pattern = '[a-zA-Z]+(?=,)')
# train.imp$FamSize <- paste0(train.imp$TotalFam,train.imp$LastName)
# train.imp$LastName <- NULL
train.imp$Name <- NULL
train.imp$LargeParCh <- as.numeric(train.imp$Parch>=3)
train.imp$LargeSibSp <- as.numeric(train.imp$SibSp>=3)
train.imp$Single <- ifelse(train.imp$TotalFam==1,1,0)
train.imp$Couple <- ifelse(train.imp$TotalFam==2,1,0)
train.imp$Family <- ifelse(train.imp$TotalFam>2,1,0)

Cabin related: Extracting the cabin alphabet and number from the cabin variable. Since the cabin numbers could be ordered from left to right or top to bottom on the boat, perhaps only the 1st digit is significant. Also, some folks have more than 1 cabin. Wonder if that’s important. Since lots of unknowns in the Cabin variable, all NA values are replaced by ‘U’. Refering to the deck diagram, the topmost decks are A and B, which are closest to the lifeboats. Perhaps that’s important too. Here, I create a bunch of categorical variables based off the original Cabin, and then remove it from the dataset.

train.imp$CabinMissing <- as.numeric(is.na(train.raw$Cabin))
train.imp$CabinCode <- map_chr(train.raw$Cabin,~str_split(string = .x,pattern = '')[[1]][1])
train.imp$CabinCode[is.na(train.imp$CabinCode)] <- 'U'
train.imp$CabinNum <- as.numeric(map_chr(train.raw$Cabin,~str_split(string = .x,pattern = '[a-zA-Z]')[[1]][2]))
train.imp$CabinNum <- map_int(train.imp$CabinNum, ~as.integer(str_split(.x,pattern = '',simplify = T)[1][1]))
train.imp$CabinNum[is.na(train.imp$CabinNum)] <- 0
train.imp$TopDeck <- ifelse(train.imp$CabinCode %in% c('A','B'),1,0)
train.imp$MidDeck <- ifelse(train.imp$CabinCode %in% c('C','D'),1,0)
train.imp$LowerDeck <- ifelse(train.imp$TopDeck==0 & train.imp$MidDeck ==0 ,1,0)
train.imp$NumberofCabins <- map_int(train.raw$Cabin,~str_split(string = .x,pattern = ' ')[[1]] %>% length)
train.imp$Cabin <- NULL

Ticket: Lastly, the ticket variable. I’m not sure what to make of it, so I’m keeping it for now, after cleaning it up a bit. A majority (80%) of the rows have unique (one) ticket. 14% rows have a duplicate ticket, perhaps indicating a family. A small number of rows have 3+ duplicates of the tickets.

train.imp$Ticket %>% table() %>% as.numeric() %>% table()

.
  1   2   3   4   5   6   7 
547  94  21  11   2   3   3 

There seems to be a bit of a pattern here. Tickets starting with 1 are mostly 1st class, those starting with 2 are 2nd class, and 3 - 3rd class. But, I feel it’s a very loose association.

train.imp %>% group_by(Pclass) %>% dplyr::select(Ticket,Pclass) %>% sample_n(5)

What I’m going to do is clean up the columns (remove special characters, spaces etc), then split the Ticket column into four: TicketChar, TicketNum,TicketNumLength, TicketNumStart. (Upon running the script a few times, I’ve decided to get rid of TicketNum, but I’m commenting the code for future ref). The TicketChar variable as this distribution:

train.imp %<>%
    mutate(
        Ticket = str_to_upper(Ticket) %>%
            str_replace_all(pattern = regex(pattern = '[.\\/]'),replacement = ''),
        TicketNum = str_extract(Ticket,pattern = regex('([0-9]){3,}')),
        TicketNumStart = map_int(TicketNum,~as.integer(str_split(.x,pattern = '',simplify = T)[1])),
        TicketNumLen = map_int(TicketNum,~dim(str_split(.x,pattern = '',simplify = T))[2]),
        TicketChar = str_extract(Ticket,pattern = regex('^[a-zA-Z/\\.]+')) 
        ) %>%
     mutate(
         TicketChar = map_chr(.x=TicketChar,
                              .f=~str_split(string=.x, pattern = '',simplify = T)[1])
         ) %>%     
    mutate(
        TicketChar = ifelse(is.na(TicketChar),'U',TicketChar),
        TicketNumStart = ifelse(is.na(TicketNumStart),0,TicketNumStart),
        TicketNumLen = ifelse(is.na(TicketNumLen),0,TicketNumLen),
    )
train.imp$Ticket <- NULL
train.imp$TicketNum <- NULL
table(train.imp$TicketChar)

  A   C   F   L   P   S   U   W 
 29  47   7   4  65  65 661  13 

table(train.imp$TicketNumLen)

  1   3   4   5   6   7 
  6   7 165 246 423  44 

table(train.imp$TicketNumStart)

  0   1   2   3   4   5   6   7   8   9 
  6 231 230 365  15   9  14  15   3   3 

Winzoring Variables

The fare variable has one massive outlier. Winzorising this variable using the 95th percentile value as the cutoff.

ggplot(train.imp,aes(x=Fare,fill=Pclass))+geom_histogram()+facet_grid(Pclass~.)

quantile(train.imp$Fare[train.imp$Pclass=='P1'],probs = c(.1,.25,.5,.75,.95))

      10%       25%       50%       75%       95% 
 26.55000  30.92395  60.28750  93.50000 232.52395 

train.imp$Fare[train.imp$Fare>232] <- 232

Final Data Review

The dataset is now prepared for modeling. Here’s a quick review of the data so far. 29 variables in total.

train.imp %>% glimpse()

Observations: 891
Variables: 29
$ Survived       <fctr> Dead, Survived, Survived, Survived, Dead, Dead, Dead, Dead, Survive...
$ Pclass         <fctr> P3, P1, P3, P1, P3, P3, P1, P3, P3, P2, P3, P1, P3, P3, P3, P2, P3,...
$ Sex            <fctr> male, female, female, female, male, male, male, male, female, femal...
$ Age            <dbl> 22.00000, 38.00000, 26.00000, 35.00000, 35.00000, 35.04936, 54.00000...
$ SibSp          <int> 1, 1, 0, 1, 0, 0, 0, 3, 0, 1, 1, 0, 0, 1, 0, 0, 4, 0, 1, 0, 0, 0, 0,...
$ Parch          <int> 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 1, 0, 0, 5, 0, 0, 1, 0, 0, 0, 0, 0, 0,...
$ Fare           <dbl> 7.2500, 71.2833, 7.9250, 53.1000, 8.0500, 8.4583, 51.8625, 21.0750, ...
$ Embarked       <fctr> S, C, S, S, S, Q, S, S, S, C, S, S, S, S, S, S, Q, S, S, C, S, S, Q...
$ title          <fctr> Mr, Mrs, Miss, Mrs, Mr, Mr, Mr, Master, Mrs, Mrs, Miss, Miss, Mr, M...
$ child          <dbl> 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1,...
$ almostadult    <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
$ Young          <dbl> 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,...
$ Seniors        <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
$ TotalFam       <dbl> 2, 2, 1, 2, 1, 1, 1, 5, 3, 2, 3, 1, 1, 7, 1, 1, 6, 1, 2, 1, 1, 1, 1,...
$ LargeParCh     <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
$ LargeSibSp     <dbl> 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,...
$ Single         <dbl> 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1,...
$ Couple         <dbl> 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,...
$ Family         <dbl> 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0,...
$ CabinMissing   <dbl> 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1,...
$ CabinCode      <chr> "U", "C", "U", "C", "U", "U", "E", "U", "U", "U", "G", "C", "U", "U"...
$ CabinNum       <dbl> 0, 8, 0, 1, 0, 0, 4, 0, 0, 0, 6, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0,...
$ TopDeck        <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
$ MidDeck        <dbl> 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,...
$ LowerDeck      <dbl> 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1,...
$ NumberofCabins <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,...
$ TicketNumStart <dbl> 2, 1, 3, 1, 3, 3, 1, 3, 3, 2, 9, 1, 2, 3, 3, 2, 3, 2, 3, 2, 2, 2, 3,...
$ TicketNumLen   <int> 5, 5, 7, 6, 6, 6, 5, 6, 6, 6, 4, 6, 4, 6, 6, 6, 6, 6, 6, 4, 6, 6, 6,...
$ TicketChar     <chr> "A", "P", "S", "U", "U", "U", "U", "U", "U", "U", "P", "U", "A", "U"...

---

Modeling

I’m experimenting with a few modeling techniques, mainly xgboost, gbm, and penalized models using glmnet. I’ve implemented all these models using caret which I find an absolutely indispensible toolkit to prep, build, tune and explore numerous models using very few lines of code.

For all models, I’m using a 5-repeat 10-fold cross validation technique on the training dataset. Thus, I have not split the training dataset further into test-train sets, given the small number of observations in the dataset.

Furthermore, given the 80:20 class-imbalance, I’m also trying out smote as an class balancing technique for a few models.

Tuning parameter searches (aka hypertuning) is performed using the tuneGrid parameter in the train() call. The best model is selected using the AUC of the ROC. Here are the models and a few intermediate results for each model. At the end, I’ve compared the performance of all the models together.

Extreme Gradient Boosting (xgboost)

xgbFit

eXtreme Gradient Boosting 

891 samples
 53 predictor
  2 classes: 'Survived', 'Dead' 

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 801, 803, 802, 802, 802, 801, ... 
Resampling results across tuning parameters:

  eta  max_depth  nrounds  ROC        Sens       Spec     
  0.3  2          2        0.8581505  0.6148067  0.9052660
  0.3  2          3        0.8677307  0.7174454  0.8699394
  0.3  2          4        0.8679025  0.7310084  0.8728485
  0.3  2          5        0.8704769  0.7262689  0.8768687
  0.3  2          6        0.8712057  0.7315630  0.8710303
  0.3  2          7        0.8718286  0.7338655  0.8706667
  0.3  3          2        0.8737598  0.7373782  0.8702963
  0.3  3          3        0.8771590  0.7403025  0.8750303
  0.3  3          4        0.8790200  0.7339160  0.8775623
  0.3  3          5        0.8804902  0.7333109  0.8815690
  0.3  3          6        0.8806026  0.7350420  0.8808620
  0.3  3          7        0.8807379  0.7256975  0.8856027
  0.3  4          2        0.8768955  0.7425882  0.8757710
  0.3  4          3        0.8805986  0.7373277  0.8794141
  0.3  4          4        0.8816917  0.7432101  0.8801145
  0.3  4          5        0.8826447  0.7444538  0.8819259
  0.3  4          6        0.8815038  0.7402185  0.8812121
  0.3  4          7        0.8804470  0.7273109  0.8833872
  0.3  5          2        0.8760521  0.7307059  0.8706397
  0.3  5          3        0.8774392  0.7290084  0.8706532
  0.3  5          4        0.8802163  0.7325210  0.8797845
  0.3  5          5        0.8793293  0.7395294  0.8775960
  0.3  5          6        0.8806162  0.7360336  0.8797778
  0.3  5          7        0.8828496  0.7377143  0.8797778
  0.3  6          2        0.8765941  0.7447059  0.8673535
  0.3  6          3        0.8772791  0.7359832  0.8706599
  0.3  6          4        0.8809166  0.7342857  0.8819057
  0.3  6          5        0.8833376  0.7430084  0.8804646
  0.3  6          6        0.8819891  0.7401008  0.8837508
  0.3  6          7        0.8839215  0.7388908  0.8844916
  0.3  7          2        0.8762051  0.7331092  0.8761010
  0.3  7          3        0.8775272  0.7423866  0.8812189
  0.3  7          4        0.8798993  0.7394286  0.8830370
  0.3  7          5        0.8803574  0.7435798  0.8852323
  0.3  7          6        0.8816307  0.7423866  0.8855960
  0.3  7          7        0.8818346  0.7429748  0.8918047
  0.5  2          2        0.8575588  0.6287899  0.8965320
  0.5  2          3        0.8669722  0.7303529  0.8714074
  0.5  2          4        0.8700492  0.7315126  0.8680943
  0.5  2          5        0.8704436  0.7315126  0.8721145
  0.5  2          6        0.8750264  0.7408571  0.8721145
  0.5  2          7        0.8764070  0.7454790  0.8786734
  0.5  3          2        0.8757504  0.7373950  0.8684714
  0.5  3          3        0.8780924  0.7397479  0.8695556
  0.5  3          4        0.8770632  0.7216134  0.8885320
  0.5  3          5        0.8781472  0.7327059  0.8819663
  0.5  3          6        0.8763049  0.7367227  0.8866734
  0.5  3          7        0.8769986  0.7308235  0.8874209
  0.5  4          2        0.8791924  0.7420000  0.8794074
  0.5  4          3        0.8805028  0.7425714  0.8739327
  0.5  4          4        0.8797489  0.7337983  0.8808889
  0.5  4          5        0.8792675  0.7314454  0.8856027
  0.5  4          6        0.8802721  0.7325042  0.8855960
  0.5  4          7        0.8808258  0.7366218  0.8866936
  0.5  5          2        0.8752125  0.7318319  0.8721010
  0.5  5          3        0.8778833  0.7303025  0.8775690
  0.5  5          4        0.8802324  0.7336807  0.8823030
  0.5  5          5        0.8792583  0.7296471  0.8819394
  0.5  5          6        0.8810007  0.7249412  0.8870438
  0.5  5          7        0.8802394  0.7249244  0.8844916
  0.5  6          2        0.8732269  0.7353613  0.8706465
  0.5  6          3        0.8761028  0.7377143  0.8801347
  0.5  6          4        0.8777128  0.7335462  0.8837845
  0.5  6          5        0.8791022  0.7341176  0.8841684
  0.5  6          6        0.8782975  0.7341176  0.8899731
  0.5  6          7        0.8771589  0.7323866  0.8892458
  0.5  7          2        0.8743506  0.7377815  0.8746330
  0.5  7          3        0.8772219  0.7419160  0.8874007
  0.5  7          4        0.8778936  0.7406723  0.8888687
  0.5  7          5        0.8804557  0.7441513  0.8903300
  0.5  7          6        0.8815295  0.7429580  0.8852189
  0.5  7          7        0.8810468  0.7377311  0.8877576

Tuning parameter 'gamma' was held constant at a value of 1
Tuning parameter
 was held constant at a value of 1
Tuning parameter 'subsample' was held constant at a value
 of 1
ROC was used to select the optimal model using  the largest value.
The final values used for the model were nrounds = 7, max_depth = 6, eta = 0.3, gamma =
 1, colsample_bytree = 1, min_child_weight = 1 and subsample = 1.

plot(xgbFit)
xgb.importance(feature_names = colnames(Dtrain),model = xgbFit$finalModel) %>%
    xgb.ggplot.importance()

densityplot(xgbFit,pch='|')
predict(xgbFit,type = 'prob') -> train.Probs
histogram(~Survived+Dead,train.Probs)

Extreme Gradient Boosting (xgboost) - SMOTE Sampling

ctrl <- trainControl(method = "repeatedcv",
                     repeats = 5,
                     verboseIter = T,
                     classProbs = TRUE,
                     summaryFunction = twoClassSummary,
                     sampling = 'smote'
                     )
xgbGrid <- expand.grid(
    nrounds=c(2,3,4,5,6,7),
    max_depth=c(2,3,4,5,6,7),
    eta=c(0.3,0.5),
    gamma=1,
    colsample_bytree=1,
    min_child_weight=1,
    subsample=1
)
dumV <- dummyVars(formula = Survived~.,data = train.imp)
Dtrain <- predict(dumV,train.imp)

variable 'Survived' is not a factor

xgbsmoteFit <- train(
    x=Dtrain,
    y=train.imp$Survived,
    method = 'xgbTree',
    trControl = ctrl,
    # metric = "Kappa",
    tuneGrid = xgbGrid,
    verbose = TRUE
)

The metric "Accuracy" was not in the result set. ROC will be used instead.

+ Fold01.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 

Attaching package: ‘DMwR’

The following object is masked from ‘package:plyr’:

    join

- Fold01.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep1: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep2: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep3: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep4: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold01.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold01.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold02.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold02.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold03.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold03.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold04.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold04.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold05.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold05.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold06.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold06.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold07.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold07.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold08.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold08.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold09.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold09.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.3, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.3, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.3, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.3, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.3, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.3, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.5, max_depth=2, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.5, max_depth=3, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.5, max_depth=4, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.5, max_depth=5, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.5, max_depth=6, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
+ Fold10.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
- Fold10.Rep5: eta=0.5, max_depth=7, gamma=1, colsample_bytree=1, min_child_weight=1, subsample=1, nrounds=7 
Aggregating results
Selecting tuning parameters
Fitting nrounds = 7, max_depth = 6, eta = 0.3, gamma = 1, colsample_bytree = 1, min_child_weight = 1, subsample = 1 on full training set

xgbsmoteFit

eXtreme Gradient Boosting 

891 samples
 53 predictor
  2 classes: 'Survived', 'Dead' 

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 802, 802, 802, 802, 802, 802, ... 
Addtional sampling using SMOTE

Resampling results across tuning parameters:

  eta  max_depth  nrounds  ROC        Sens       Spec     
  0.3  2          2        0.8422230  0.6017983  0.9041886
  0.3  2          3        0.8492899  0.6334454  0.9001886
  0.3  2          4        0.8559285  0.6701345  0.8968822
  0.3  2          5        0.8626094  0.6590252  0.9031044
  0.3  2          6        0.8654252  0.6521345  0.9060135
  0.3  2          7        0.8660548  0.6718824  0.9016296
  0.3  3          2        0.8594054  0.6228067  0.9235017
  0.3  3          3        0.8638679  0.5971429  0.9336768
  0.3  3          4        0.8659124  0.6034622  0.9326263
  0.3  3          5        0.8663275  0.6007563  0.9300404
  0.3  3          6        0.8692959  0.6066218  0.9307879
  0.3  3          7        0.8724934  0.6129412  0.9256700
  0.3  4          2        0.8552456  0.5870756  0.9348148
  0.3  4          3        0.8626897  0.5895462  0.9337104
  0.3  4          4        0.8694388  0.5895294  0.9358923
  0.3  4          5        0.8729766  0.5995126  0.9362559
  0.3  4          6        0.8755678  0.6064370  0.9366128
  0.3  4          7        0.8794687  0.6110252  0.9388081
  0.3  5          2        0.8600815  0.5929748  0.9329832
  0.3  5          3        0.8633800  0.5917983  0.9311515
  0.3  5          4        0.8689604  0.6000000  0.9347946
  0.3  5          5        0.8729573  0.6028739  0.9377104
  0.3  5          6        0.8751460  0.6105042  0.9340673
  0.3  5          7        0.8766301  0.6169580  0.9333199
  0.3  6          2        0.8633920  0.6239664  0.9191380
  0.3  6          3        0.8673741  0.6320504  0.9205859
  0.3  6          4        0.8715988  0.6268235  0.9231178
  0.3  6          5        0.8750872  0.6378992  0.9220471
  0.3  6          6        0.8761488  0.6419832  0.9209495
  0.3  6          7        0.8800014  0.6496134  0.9169630
  0.3  7          2        0.8561804  0.6192269  0.9195017
  0.3  7          3        0.8610940  0.6321008  0.9162222
  0.3  7          4        0.8646876  0.6397143  0.9176970
  0.3  7          5        0.8695185  0.6490924  0.9129428
  0.3  7          6        0.8710938  0.6572269  0.9143973
  0.3  7          7        0.8713630  0.6619496  0.9143906
  0.5  2          2        0.8490018  0.6370252  0.9012323
  0.5  2          3        0.8599325  0.6549748  0.9038047
  0.5  2          4        0.8629421  0.6410756  0.9125320
  0.5  2          5        0.8664901  0.6181849  0.9194815
  0.5  2          6        0.8674602  0.6101176  0.9202424
  0.5  2          7        0.8699278  0.6131933  0.9238451
  0.5  3          2        0.8610398  0.6136471  0.9242290
  0.5  3          3        0.8634444  0.5919160  0.9347744
  0.5  3          4        0.8695581  0.6042521  0.9296700
  0.5  3          5        0.8712536  0.5908571  0.9289495
  0.5  3          6        0.8726467  0.5941681  0.9347946
  0.5  3          7        0.8752367  0.6024370  0.9351515
  0.5  4          2        0.8626870  0.6024706  0.9329966
  0.5  4          3        0.8693973  0.5964034  0.9347879
  0.5  4          4        0.8755566  0.6059496  0.9315219
  0.5  4          5        0.8759294  0.6204874  0.9373401
  0.5  4          6        0.8739551  0.6158824  0.9373401
  0.5  4          7        0.8744622  0.6292269  0.9318855
  0.5  5          2        0.8631468  0.6064874  0.9278923
  0.5  5          3        0.8709386  0.6163361  0.9282357
  0.5  5          4        0.8742147  0.6315126  0.9267879
  0.5  5          5        0.8749953  0.6338992  0.9271582
  0.5  5          6        0.8747302  0.6426723  0.9238788
  0.5  5          7        0.8757683  0.6466891  0.9191515
  0.5  6          2        0.8625161  0.6239832  0.9213199
  0.5  6          3        0.8671210  0.6332605  0.9184108
  0.5  6          4        0.8676388  0.6419496  0.9111246
  0.5  6          5        0.8689798  0.6571765  0.9154949
  0.5  6          6        0.8728196  0.6530756  0.9125926
  0.5  6          7        0.8742729  0.6594958  0.9129495
  0.5  7          2        0.8649452  0.6513950  0.9213064
  0.5  7          3        0.8670273  0.6544034  0.9209562
  0.5  7          4        0.8748885  0.6684370  0.9198855
  0.5  7          5        0.8736510  0.6684034  0.9176835
  0.5  7          6        0.8763594  0.6748403  0.9180337
  0.5  7          7        0.8767229  0.6777983  0.9154949

Tuning parameter 'gamma' was held constant at a value of 1
Tuning parameter
 was held constant at a value of 1
Tuning parameter 'subsample' was held constant at a value
 of 1
ROC was used to select the optimal model using  the largest value.
The final values used for the model were nrounds = 7, max_depth = 6, eta = 0.3, gamma =
 1, colsample_bytree = 1, min_child_weight = 1 and subsample = 1.

plot(xgbsmoteFit)

xgb.importance(feature_names = colnames(Dtrain),model = xgbsmoteFit$finalModel)

xgb.importance(feature_names = colnames(Dtrain),model = xgbsmoteFit$finalModel) %>%
xgb.ggplot.importance()

densityplot(xgbsmoteFit,pch='|')

predict(xgbsmoteFit,type = 'raw') -> train.Class
predict(xgbsmoteFit,type = 'prob') -> train.Probs
histogram(~Survived+Dead,train.Probs)

Gradient Boosting (gbm)

boostFit

Stochastic Gradient Boosting 

891 samples
 53 predictor
  2 classes: 'Survived', 'Dead' 

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 803, 802, 802, 802, 802, 802, ... 
Resampling results across tuning parameters:

  shrinkage  interaction.depth  n.trees  ROC        Sens       Spec     
  0.01       1                   500     0.8687740  0.7176975  0.8812727
  0.01       1                   700     0.8715465  0.7310756  0.8827340
  0.01       1                   900     0.8731344  0.7456975  0.8798182
  0.01       1                  1100     0.8737417  0.7614454  0.8790909
  0.01       2                   500     0.8768379  0.7410252  0.8819933
  0.01       2                   700     0.8781806  0.7549916  0.8808956
  0.01       2                   900     0.8791223  0.7614118  0.8801751
  0.01       2                  1100     0.8801043  0.7613950  0.8809024
  0.01       3                   500     0.8790775  0.7526218  0.8823636
  0.01       3                   700     0.8808075  0.7584874  0.8812727
  0.01       3                   900     0.8815746  0.7549748  0.8809091
  0.01       3                  1100     0.8817104  0.7532269  0.8849091
  0.10       1                   500     0.8699484  0.7480168  0.8721751
  0.10       1                   700     0.8708636  0.7485882  0.8736162
  0.10       1                   900     0.8686151  0.7422017  0.8743367
  0.10       1                  1100     0.8689881  0.7428067  0.8725051
  0.10       2                   500     0.8801393  0.7473950  0.8837845
  0.10       2                   700     0.8799571  0.7467563  0.8819798
  0.10       2                   900     0.8790711  0.7438992  0.8790370
  0.10       2                  1100     0.8773468  0.7474118  0.8757778
  0.10       3                   500     0.8803987  0.7450252  0.8790774
  0.10       3                   700     0.8805280  0.7438655  0.8746869
  0.10       3                   900     0.8793306  0.7438655  0.8692189
  0.10       3                  1100     0.8785908  0.7432773  0.8713872

Tuning parameter 'n.minobsinnode' was held constant at a value of 10
ROC was used to select the optimal model using  the largest value.
The final values used for the model were n.trees = 1100, interaction.depth = 3, shrinkage
 = 0.01 and n.minobsinnode = 10.

plot(boostFit)

xyplot(oobag.improve~1:1100,data=boostFit$finalModel,alpha=.5,xlab = 'n.trees')

plot(varImp(boostFit))

densityplot(boostFit,pch='|')

predict(boostFit,type = 'prob') -> train.Probs
histogram(~Survived+Dead,train.Probs)

Random Forest (rf)

rfFit.y

Random Forest 

891 samples
 28 predictor
  2 classes: 'Survived', 'Dead' 

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 803, 802, 802, 802, 802, 802, ... 
Resampling results across tuning parameters:

  mtry  ROC        Sens       Spec     
   5    0.8842523  0.7122353  0.8969293
  10    0.8885360  0.7286891  0.8940202
  15    0.8874201  0.7351429  0.8878182
  20    0.8864149  0.7415462  0.8867138
  25    0.8866698  0.7427563  0.8816229

ROC was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 10.

plot(rfFit.y)

plot(rfFit.y$finalModel)

densityplot(rfFit.y,pch='|')

predict(rfFit.y,type = 'prob') -> train.rf.Probs
histogram(~Survived+Dead,train.rf.Probs)

Random Forest (rf) - SMOTE

rfsmoteFit.y

Random Forest 

891 samples
 28 predictor
  2 classes: 'Survived', 'Dead' 

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 803, 802, 802, 802, 802, 802, ... 
Addtional sampling using SMOTE

Resampling results across tuning parameters:

  mtry  ROC        Sens       Spec     
   5    0.8783818  0.6333782  0.9369966
  10    0.8821756  0.6854286  0.9129697
  15    0.8831274  0.6983025  0.9016498
  20    0.8814043  0.7129580  0.8910976
  25    0.8793606  0.7135462  0.8885455

ROC was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 15.

plot(rfsmoteFit.y)

plot(rfsmoteFit.y$finalModel)

densityplot(rfsmoteFit.y,pch='|')

predict(rfsmoteFit.y,type = 'prob') -> train.rfsmoteFit.Probs
histogram(~Survived+Dead,train.rfsmoteFit.Probs)

Elastinet

glmnetFit

glmnet 

891 samples
 53 predictor
  2 classes: 'Survived', 'Dead' 

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 5 times) 
Summary of sample sizes: 803, 802, 802, 802, 802, 802, ... 
Addtional sampling using SMOTE

Resampling results across tuning parameters:

  alpha  lambda      ROC        Sens       Spec     
  0.0    0.01000000  0.8624351  0.7269580  0.8875152
  0.0    0.01487179  0.8624351  0.7269580  0.8875152
  0.0    0.01974359  0.8624351  0.7269580  0.8875152
  0.0    0.02461538  0.8624351  0.7269580  0.8875152
  0.0    0.02948718  0.8624351  0.7269580  0.8875152
  0.0    0.03435897  0.8626287  0.7269580  0.8871515
  0.0    0.03923077  0.8629192  0.7269412  0.8871515
  0.0    0.04410256  0.8631763  0.7269076  0.8860539
  0.0    0.04897436  0.8633421  0.7263361  0.8856902
  0.0    0.05384615  0.8631602  0.7246218  0.8860539
  0.0    0.05871795  0.8632236  0.7234286  0.8856902
  0.0    0.06358974  0.8634279  0.7240168  0.8853266
  0.0    0.06846154  0.8634157  0.7240168  0.8838721
  0.0    0.07333333  0.8634152  0.7246218  0.8835017
  0.0    0.07820513  0.8633267  0.7252269  0.8838653
  0.0    0.08307692  0.8633870  0.7246723  0.8842290
  0.0    0.08794872  0.8632672  0.7264370  0.8838653
  0.0    0.09282051  0.8630221  0.7258487  0.8838653
  0.0    0.09769231  0.8630112  0.7246723  0.8835017
  0.0    0.10256410  0.8630108  0.7258487  0.8827744
  0.0    0.10743590  0.8629793  0.7270252  0.8820337
  0.0    0.11230769  0.8629057  0.7275966  0.8820337
  0.0    0.11717949  0.8629160  0.7264370  0.8816700
  0.0    0.12205128  0.8629367  0.7252605  0.8816700
  0.0    0.12692308  0.8628505  0.7246723  0.8813064
  0.0    0.13179487  0.8627742  0.7252437  0.8813064
  0.0    0.13666667  0.8626347  0.7240840  0.8816700
  0.0    0.14153846  0.8626660  0.7234790  0.8816700
  0.0    0.14641026  0.8624861  0.7228908  0.8816700
  0.0    0.15128205  0.8623359  0.7205378  0.8809360
  0.0    0.15615385  0.8622288  0.7211261  0.8805724
  0.0    0.16102564  0.8621643  0.7211092  0.8798451
  0.0    0.16589744  0.8620677  0.7199496  0.8791111
  0.0    0.17076923  0.8620130  0.7193782  0.8787475
  0.0    0.17564103  0.8620221  0.7193782  0.8791111
  0.0    0.18051282  0.8620338  0.7193782  0.8791111
  0.0    0.18538462  0.8620142  0.7193782  0.8780202
  0.0    0.19025641  0.8620445  0.7182017  0.8765657
  0.0    0.19512821  0.8619043  0.7170252  0.8765657
  0.0    0.20000000  0.8618487  0.7152773  0.8758384
  0.1    0.01000000  0.8600372  0.7241176  0.8878519
  0.1    0.01487179  0.8615477  0.7264538  0.8896768
  0.1    0.01974359  0.8624891  0.7241345  0.8875017
  0.1    0.02461538  0.8631904  0.7223866  0.8878653
  0.1    0.02948718  0.8635738  0.7270252  0.8871246
  0.1    0.03435897  0.8640134  0.7281513  0.8867542
  0.1    0.03923077  0.8642901  0.7257983  0.8860269
  0.1    0.04410256  0.8644447  0.7257983  0.8852997
  0.1    0.04897436  0.8644893  0.7246218  0.8852997
  0.1    0.05384615  0.8647542  0.7234622  0.8845724
  0.1    0.05871795  0.8648624  0.7205378  0.8849360
  0.1    0.06358974  0.8646418  0.7205378  0.8849360
  0.1    0.06846154  0.8645368  0.7211261  0.8849360
  0.1    0.07333333  0.8643896  0.7193950  0.8842020
  0.1    0.07820513  0.8643594  0.7176303  0.8842020
  0.1    0.08307692  0.8642748  0.7141008  0.8845589
  0.1    0.08794872  0.8640778  0.7135294  0.8841953
  0.1    0.09282051  0.8636771  0.7123697  0.8834815
  0.1    0.09769231  0.8635362  0.7129580  0.8823906
  0.1    0.10256410  0.8631628  0.7123697  0.8805657
  0.1    0.10743590  0.8628222  0.7123529  0.8798316
  0.1    0.11230769  0.8625770  0.7106050  0.8794680
  0.1    0.11717949  0.8622052  0.7111933  0.8780135
  0.1    0.12205128  0.8619550  0.7111933  0.8772795
  0.1    0.12692308  0.8616472  0.7100336  0.8765522
  0.1    0.13179487  0.8616144  0.7088908  0.8761886
  0.1    0.13666667  0.8614748  0.7083025  0.8747340
  0.1    0.14153846  0.8612211  0.7082857  0.8747340
  0.1    0.14641026  0.8611024  0.7076975  0.8743636
  0.1    0.15128205  0.8609678  0.7065210  0.8740000
  0.1    0.15615385  0.8605709  0.7059328  0.8732727
  0.1    0.16102564  0.8604414  0.7059496  0.8725455
  0.1    0.16589744  0.8602385  0.7048067  0.8721751
  0.1    0.17076923  0.8599341  0.7036471  0.8725387
  0.1    0.17564103  0.8598033  0.7042353  0.8718114
  0.1    0.18051282  0.8595210  0.7036639  0.8707205
  0.1    0.18538462  0.8591656  0.7024874  0.8703569
  0.1    0.19025641  0.8589091  0.7018992  0.8699933
  0.1    0.19512821  0.8584617  0.7013277  0.8703569
  0.1    0.20000000  0.8581420  0.7013277  0.8692660
  0.2    0.01000000  0.8591220  0.7280672  0.8882020
  0.2    0.01487179  0.8604820  0.7245882  0.8903906
  0.2    0.01974359  0.8618651  0.7228571  0.8911178
  0.2    0.02461538  0.8620406  0.7228908  0.8892997
  0.2    0.02948718  0.8620432  0.7217143  0.8889360
  0.2    0.03435897  0.8622358  0.7234958  0.8889428
  0.2    0.03923077  0.8622652  0.7223361  0.8885859
  0.2    0.04410256  0.8620627  0.7229412  0.8889495
  0.2    0.04897436  0.8620116  0.7188235  0.8874949
  0.2    0.05384615  0.8619238  0.7153109  0.8864040
  0.2    0.05871795  0.8619083  0.7147227  0.8856700
  0.2    0.06358974  0.8614219  0.7141513  0.8853064
  0.2    0.06846154  0.8613769  0.7094790  0.8845724
  0.2    0.07333333  0.8613267  0.7089076  0.8849360
  0.2    0.07820513  0.8613301  0.7071429  0.8842088
  0.2    0.08307692  0.8613348  0.7065546  0.8823906
  0.2    0.08794872  0.8612440  0.7036471  0.8805724
  0.2    0.09282051  0.8607954  0.7036639  0.8791111
  0.2    0.09769231  0.8604293  0.7007563  0.8765657
  0.2    0.10256410  0.8599402  0.7007563  0.8747475
  0.2    0.10743590  0.8591161  0.7025378  0.8736431
  0.2    0.11230769  0.8580974  0.7019496  0.8721818
  0.2    0.11717949  0.8576971  0.7025042  0.8710774
  0.2    0.12205128  0.8571679  0.7025042  0.8703502
  0.2    0.12692308  0.8567174  0.7024874  0.8688956
  0.2    0.13179487  0.8561708  0.7007395  0.8656229
  0.2    0.13666667  0.8550908  0.7013277  0.8648956
  0.2    0.14153846  0.8544119  0.7007395  0.8645320
  0.2    0.14641026  0.8540218  0.7007563  0.8641684
  0.2    0.15128205  0.8534856  0.7007563  0.8638047
  0.2    0.15615385  0.8528506  0.7001681  0.8627138
  0.2    0.16102564  0.8523328  0.6984034  0.8627138
  0.2    0.16589744  0.8514521  0.6984034  0.8619865
  0.2    0.17076923  0.8508972  0.6978151  0.8598047
  0.2    0.17564103  0.8506594  0.6972269  0.8594411
  0.2    0.18051282  0.8500216  0.6978151  0.8583434
  0.2    0.18538462  0.8495585  0.6972437  0.8572458
  0.2    0.19025641  0.8490212  0.6966723  0.8561549
  0.2    0.19512821  0.8488122  0.6955126  0.8557912
  0.2    0.20000000  0.8477981  0.6949244  0.8547003
  0.4    0.01000000  0.8608629  0.7170924  0.8856835
  0.4    0.01487179  0.8630875  0.7188235  0.8875017
  0.4    0.01974359  0.8635522  0.7235126  0.8882290
  0.4    0.02461538  0.8638852  0.7241008  0.8863973
  0.4    0.02948718  0.8640961  0.7246387  0.8874882
  0.4    0.03435897  0.8636815  0.7205882  0.8878519
  0.4    0.03923077  0.8634645  0.7188403  0.8867542
  0.4    0.04410256  0.8625804  0.7165210  0.8845589
  0.4    0.04897436  0.8610777  0.7129916  0.8845589
  0.4    0.05384615  0.8603462  0.7106218  0.8812727
  0.4    0.05871795  0.8593766  0.7100168  0.8783569
  0.4    0.06358974  0.8586905  0.7106387  0.8758047
  0.4    0.06846154  0.8577189  0.7083193  0.8739865
  0.4    0.07333333  0.8561190  0.7071765  0.8707138
  0.4    0.07820513  0.8539432  0.7042521  0.8692525
  0.4    0.08307692  0.8520797  0.7048571  0.8674276
  0.4    0.08794872  0.8510941  0.7054454  0.8652391
  0.4    0.09282051  0.8498438  0.7060168  0.8612391
  0.4    0.09769231  0.8489821  0.7042521  0.8590572
  0.4    0.10256410  0.8485292  0.7054286  0.8543232
  0.4    0.10743590  0.8478570  0.7042521  0.8535960
  0.4    0.11230769  0.8474843  0.7013277  0.8514074
  0.4    0.11717949  0.8474952  0.7001345  0.8514141
  0.4    0.12205128  0.8472912  0.6995462  0.8503232
  0.4    0.12692308  0.8465502  0.6989580  0.8499596
  0.4    0.13179487  0.8458615  0.6977983  0.8495960
  0.4    0.13666667  0.8460465  0.6960336  0.8499596
  0.4    0.14153846  0.8456487  0.6936807  0.8510640
  0.4    0.14641026  0.8457106  0.6919328  0.8510640
  0.4    0.15128205  0.8463243  0.6884034  0.8514276
  0.4    0.15615385  0.8466792  0.6872269  0.8514276
  0.4    0.16102564  0.8467969  0.6860840  0.8514276
  0.4    0.16589744  0.8465443  0.6860840  0.8517912
  0.4    0.17076923  0.8460047  0.6854958  0.8521549
  0.4    0.17564103  0.8456889  0.6843193  0.8521549
  0.4    0.18051282  0.8452321  0.6843193  0.8521549
  0.4    0.18538462  0.8443937  0.6831765  0.8525185
  0.4    0.19025641  0.8439948  0.6831765  0.8525185
  0.4    0.19512821  0.8432182  0.6831765  0.8525185
  0.4    0.20000000  0.8429886  0.6820000  0.8525185
  0.6    0.01000000  0.8615146  0.7217311  0.8889428
  0.6    0.01487179  0.8625244  0.7234118  0.8911380
  0.6    0.01974359  0.8627385  0.7234622  0.8911246
  0.6    0.02461538  0.8630850  0.7211765  0.8900202
  0.6    0.02948718  0.8622451  0.7194286  0.8860000
  0.6    0.03435897  0.8620347  0.7135966  0.8830842
  0.6    0.03923077  0.8608737  0.7077311  0.8808956
  0.6    0.04410256  0.8595263  0.7077647  0.8776229
  0.6    0.04897436  0.8580028  0.7071765  0.8750774
  0.6    0.05384615  0.8563695  0.7054454  0.8725253
  0.6    0.05871795  0.8546962  0.7048739  0.8641616
  0.6    0.06358974  0.8528297  0.7066050  0.8601549
  0.6    0.06846154  0.8511820  0.7071765  0.8568822
  0.6    0.07333333  0.8498130  0.7089244  0.8521481
  0.6    0.07820513  0.8489291  0.7083361  0.8488687
  0.6    0.08307692  0.8492258  0.7071597  0.8488620
  0.6    0.08794872  0.8491236  0.7054118  0.8488620
  0.6    0.09282051  0.8492666  0.7018824  0.8488620
  0.6    0.09769231  0.8486469  0.7001176  0.8492256
  0.6    0.10256410  0.8485608  0.6954790  0.8506869
  0.6    0.10743590  0.8486213  0.6937143  0.8506869
  0.6    0.11230769  0.8478946  0.6931261  0.8506869
  0.6    0.11717949  0.8466370  0.6901849  0.8506869
  0.6    0.12205128  0.8461776  0.6860840  0.8517912
  0.6    0.12692308  0.8455451  0.6860840  0.8517912
  0.6    0.13179487  0.8436514  0.6854958  0.8517912
  0.6    0.13666667  0.8430642  0.6837647  0.8521549
  0.6    0.14153846  0.8423842  0.6831765  0.8521549
  0.6    0.14641026  0.8418510  0.6825882  0.8525185
  0.6    0.15128205  0.8407780  0.6820000  0.8525185
  0.6    0.15615385  0.8404541  0.6814118  0.8525185
  0.6    0.16102564  0.8401007  0.6814118  0.8525185
  0.6    0.16589744  0.8396466  0.6814118  0.8525185
  0.6    0.17076923  0.8391832  0.6814118  0.8525185
  0.6    0.17564103  0.8392284  0.6814118  0.8525185
  0.6    0.18051282  0.8390840  0.6814118  0.8525185
  0.6    0.18538462  0.8388220  0.6814118  0.8525185
  0.6    0.19025641  0.8389099  0.6814118  0.8525185
  0.6    0.19512821  0.8367532  0.6814118  0.8525185
  0.6    0.20000000  0.8350971  0.6814118  0.8525185
 [ reached getOption("max.print") -- omitted 80 rows ]

ROC was used to select the optimal model using  the largest value.
The final values used for the model were alpha = 0.1 and lambda = 0.05871795.

plot(glmnetFit,plotType='level')

plot(varImp(glmnetFit))

densityplot(glmnetFit,pch='|')

predict(glmnetFit,type = 'prob') -> train.glmnet.Probs
histogram(~Survived+Dead,train.glmnet.Probs)

Compare models

re <-
    resamples(
    x = list(
    xgb = xgbFit,
    xgbsmote = xgbsmoteFit,
    rf = rfFit.y,
    rfsmote = rfsmoteFit.y,
    gbm = boostFit,
    elastinet=glmnetFit
    )
    )
summary(re)

Call:
summary.resamples(object = re)

Models: xgb, xgbsmote, rf, rfsmote, gbm, elastinet 
Number of resamples: 50 

ROC 
               Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
xgb       0.7360963 0.8603304 0.8925134 0.8839215 0.9196505 0.9540260    0
xgbsmote  0.7462567 0.8653743 0.8867647 0.8800014 0.9078877 0.9350267    0
rf        0.7807487 0.8622670 0.8984301 0.8885360 0.9126833 0.9556150    0
rfsmote   0.7759358 0.8581818 0.8880026 0.8831274 0.9106551 0.9574866    0
gbm       0.7604278 0.8570834 0.8848587 0.8817104 0.9055828 0.9609626    0
elastinet 0.7518717 0.8413102 0.8650822 0.8648624 0.9010538 0.9636364    0

Sens 
               Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
xgb       0.5588235 0.7058824 0.7352941 0.7388908 0.7941176 0.9142857    0
xgbsmote  0.4411765 0.5882353 0.6470588 0.6496134 0.7058824 0.8529412    0
rf        0.5294118 0.6764706 0.7352941 0.7286891 0.7941176 0.8823529    0
rfsmote   0.5294118 0.6470588 0.6857143 0.6983025 0.7647059 0.9117647    0
gbm       0.5588235 0.6907563 0.7647059 0.7532269 0.8235294 0.9117647    0
elastinet 0.5294118 0.6470588 0.7390756 0.7205378 0.7714286 0.8529412    0

Spec 
               Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
xgb       0.7777778 0.8709596 0.8898990 0.8844916 0.9090909 0.9636364    0
xgbsmote  0.7818182 0.8909091 0.9272727 0.9169630 0.9454545 1.0000000    0
rf        0.8181818 0.8727273 0.8909091 0.8940202 0.9259259 0.9818182    0
rfsmote   0.8181818 0.8727273 0.8909091 0.9016498 0.9272727 0.9818182    0
gbm       0.8181818 0.8545455 0.8898990 0.8849091 0.9217172 0.9818182    0
elastinet 0.8000000 0.8545455 0.8909091 0.8849360 0.9090909 0.9636364    0

bwplot(re)

summary(diff(re))

Call:
summary.diff.resamples(object = diff(re))

p-value adjustment: bonferroni 
Upper diagonal: estimates of the difference
Lower diagonal: p-value for H0: difference = 0

ROC 
          xgb       xgbsmote   rf         rfsmote    gbm        elastinet 
xgb                  0.0039200 -0.0046145  0.0007941  0.0022111  0.0190591
xgbsmote  1.0000000            -0.0085345 -0.0031259 -0.0017090  0.0151390
rf        1.0000000 1.0000000              0.0054086  0.0068256  0.0236736
rfsmote   1.0000000 1.0000000  0.0511756              0.0014169  0.0182650
gbm       1.0000000 1.0000000  0.2458148  1.0000000              0.0168480
elastinet 0.7953846 1.0000000  2.206e-06  0.0007768  3.211e-05            

Sens 
          xgb       xgbsmote  rf        rfsmote   gbm       elastinet
xgb                  0.089277  0.010202  0.040588 -0.014336  0.018353
xgbsmote  0.0001186           -0.079076 -0.048689 -0.103613 -0.070924
rf        1.0000000 0.0001066            0.030387 -0.024538  0.008151
rfsmote   0.4046800 0.0365380 0.0001694           -0.054924 -0.022235
gbm       1.0000000 1.143e-06 0.0174450 9.138e-07            0.032689
elastinet 1.0000000 0.0036014 1.0000000 0.4417908 3.796e-05          

Spec 
          xgb      xgbsmote   rf         rfsmote    gbm        elastinet 
xgb                -3.247e-02 -9.529e-03 -1.716e-02 -4.175e-04 -4.444e-04
xgbsmote  0.014850             2.294e-02  1.531e-02  3.205e-02  3.203e-02
rf        1.000000 0.436352              -7.630e-03  9.111e-03  9.084e-03
rfsmote   0.503523 1.000000   0.229579               1.674e-02  1.671e-02
gbm       1.000000 0.027924   0.354081   0.007644              -2.694e-05
elastinet 1.000000 0.029234   1.000000   0.068587   1.000000             

simulatedTrain <- data.frame(Class = train.imp$Survived)
simulatedTrain$rf = predict(rfFit.y,type = 'prob')[[1]]
simulatedTrain$rfsmote = predict(rfsmoteFit.y,type = 'prob')[[1]]
simulatedTrain$xgb = predict(xgbFit,type = 'prob')[[1]]
simulatedTrain$xgbsmote = predict(xgbsmoteFit,type = 'prob')[[1]]
simulatedTrain$boost = predict(boostFit,type = 'prob')[[1]]

calCurve <- calibration(x = Class~rf+rfsmote+xgb+xgbsmote+boost,data = simulatedTrain)
xyplot(calCurve,auto.key=list(columns=3))

Calibrating probabilities

boostsigmoidCal <- glm(relevel(Class,ref='Dead')~boost,simulatedTrain,family = 'binomial')
coef(summary(boostsigmoidCal))

             Estimate Std. Error   z value     Pr(>|z|)
(Intercept) -3.393991  0.1992172 -17.03664 4.392812e-65
boost        7.063846  0.4064620  17.37886 1.193056e-67

simulatedTrain$boostSig = predict(boostsigmoidCal,type = 'response')
xgbsmotesigmoidCal <- glm(relevel(Class,ref='Dead')~xgbsmote,simulatedTrain,family = 'binomial')
coef(summary(xgbsmotesigmoidCal))

             Estimate Std. Error   z value     Pr(>|z|)
(Intercept) -3.468413  0.2016514 -17.20004 2.653386e-66
xgbsmote     9.205386  0.5945264  15.48356 4.479596e-54

simulatedTrain$xgbsmoteSig = predict(xgbsmotesigmoidCal,type = 'response')
calibration(x = Class~boost+boostSig,data = simulatedTrain) %>%
    xyplot(auto.key=list(columns=2))

calibration(x = Class~xgbsmote+xgbsmoteSig,data = simulatedTrain) %>%
    xyplot(auto.key=list(columns=2))

Evaluation

test.imp <- test.raw

#Embarked
test.imp$Embarked[is.na(test.imp$Embarked)]='S'

#Title
test.raw$title <- str_extract(pattern = '[a-zA-Z]+(?=\\.)',string = test.raw$Name)
#test.raw$title <- as.factor(test.raw$title)
test.imp$title <- as.character(test.raw$title)
test.imp$title[test.imp$title %in% c('Capt','Col','Major')] <- 'Officer'
test.imp$title[test.imp$title %in% c('Don','Dr','Rev','Sir','Jonkheer','Countess','Lady','Dona')] <- 'Royalty'
test.imp$title[test.imp$title %in% c('Mrs','Mme')] <- 'Mrs'
test.imp$title[test.imp$title %in% c('Ms','Mlle')] <- 'Miss'
test.imp$title <- as.factor(test.imp$title)

#Missing age
missing.age <- test.imp %>% filter(is.na(Age))
age.predicted <- predict(rfFit, newdata = missing.age)
test.imp$Age[is.na(test.imp$Age)] <- age.predicted
test.imp$Age[test.imp$title=='Master' & test.imp$Age > 20] <- 4

#Child
test.imp$child <- 0
test.imp$child[test.imp$Age<18] <- 1
test.imp$almostadult <- as.numeric(between(test.imp$Age,16,18))

#Young/old
test.imp$Young <- ifelse(test.imp$Age<10,1,0)
test.imp$Seniors <- ifelse(test.imp$Age>60,1,0)

#Family Related
test.imp$TotalFam <- test.imp$SibSp + test.imp$Parch + 1
test.imp$Name <- NULL
test.imp$LargeParCh <- as.numeric(test.imp$Parch>=3)
test.imp$LargeSibSp <- as.numeric(test.imp$SibSp>=3)
test.imp$Single <- ifelse(test.imp$TotalFam==1,1,0)
test.imp$Couple <- ifelse(test.imp$TotalFam==2,1,0)
test.imp$Family <- ifelse(test.imp$TotalFam>2,1,0)

#Cabin & Deck
test.imp$CabinMissing <- as.numeric(is.na(test.raw$Cabin))
test.imp$CabinCode <- map_chr(test.raw$Cabin,~str_split(string = .x,pattern = '')[[1]][1])
test.imp$CabinCode[is.na(test.imp$CabinCode)] <- 'U'
test.imp$CabinNum <- as.numeric(map_chr(test.raw$Cabin,~str_split(string = .x,pattern = '[a-zA-Z]')[[1]][2]))
test.imp$CabinNum <- map_int(test.imp$CabinNum, ~as.integer(str_split(.x,pattern = '',simplify = T)[1][1]))
test.imp$CabinNum[is.na(test.imp$CabinNum)] <- 0

test.imp$CabinCode <- factor(
    x = test.imp$CabinCode,
    levels = unique(train.imp$CabinCode)
)

test.imp$TopDeck <- ifelse(test.imp$CabinCode %in% c('A','B'),1,0)
test.imp$MidDeck <- ifelse(test.imp$CabinCode %in% c('C','D'),1,0)
test.imp$LowerDeck <- ifelse(test.imp$TopDeck==0 & test.imp$MidDeck ==0 ,1,0)

test.imp$NumberofCabins <- map_int(test.raw$Cabin,~str_split(string = .x,pattern = ' ')[[1]] %>% length)
test.imp$Cabin <- NULL


# Ticket
test.imp %<>%
    mutate(
      Ticket = str_to_upper(Ticket) %>%
          str_replace_all(pattern = regex(pattern = '[.\\/]'),replacement = ''),
      TicketNum = str_extract(Ticket,pattern = regex('([0-9]){3,}')),
      TicketNumStart = map_int(TicketNum,~as.integer(str_split(.x,pattern = '',simplify = T)[1])),
      TicketNumLen = map_int(TicketNum,~dim(str_split(.x,pattern = '',simplify = T))[2]),
      TicketChar = str_extract(Ticket,pattern = regex('^[a-zA-Z/\\.]+'))
      ) %>%
    mutate(
        TicketChar = map_chr(.x=TicketChar,
                             .f=~str_split(string=.x, pattern = '',simplify = T)[1])
        ) %>%  
    mutate(
      TicketChar = ifelse(is.na(TicketChar),'U',TicketChar),
      TicketNumStart = ifelse(is.na(TicketNumStart),0,TicketNumStart),
      TicketNumLen = ifelse(is.na(TicketNumLen),0,TicketNumLen),
    )
test.imp$Ticket <- NULL
test.imp$TicketNum <- NULL

#Fare
test.imp$Fare[is.na(test.imp$Fare)] <- 14.4542
test.imp$Fare[test.imp$Fare>232] <- 232

xgbsmotesigmoidCal

Call:  glm(formula = relevel(Class, ref = "Dead") ~ xgbsmote, family = "binomial", 
    data = simulatedTrain)

Coefficients:
(Intercept)     xgbsmote  
     -3.468        9.205  

Degrees of Freedom: 890 Total (i.e. Null);  889 Residual
Null Deviance:      1187 
Residual Deviance: 491.9    AIC: 495.9

PID <-
    readData(Titanic.path,
    test.data.file,
    test.column.types,
    missing.types)
PID <- PID$PassengerId

for(m in ls(pattern = 'Pred')) {
    write.csv(
    x = data.frame(
    PassengerId = PID,
    Survived = as.numeric(eval(parse(text = m))) * -1 + 2
    ),
    file = paste0(m,'.csv'),
    row.names = F
    )
}

Conclusions

---

1. I think this approach depends on the academic background and the industry of the analyst. Prof Srinivasan, and my mentor at work both have strong statistical academic backgrounds, and both believe in thorough EDA of the data. I’ve also noticed this approach from individuals in the banking & insurance industry - perhaps due to regulatory requirements. On the other hand, folks trained in computer science and algorithmic data science tend to underplay the importance of thorough EDA.↩

2. To iterate variable names in ggplot, use ggplot(...)+aes_string(...) in place of ggplot(...,aes(...)).↩

3. Read more about beanplots here: https://cran.r-project.org/web/packages/beanplot/vignettes/beanplot.pdf↩